Information processing apparatus, display control method, and program

ABSTRACT

There is provided an information processing apparatus including a judgment unit for judging an anteroposterior relationship between a shot actual object and a virtual object for each part by use of depth information, and a display control unit for displaying a virtual image in which the virtual object is projected to be overlapped on a shot image in which the actual object is shot based on the anteroposterior relationship judged by the judgment unit.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus, adisplay control method and a program.

BACKGROUND ART

There are proposed, as virtual try-on systems, various try-on imagegeneration techniques for combining a shot image shooting a user thereinwith a clothes image.

For example, Patent Literature 1 discloses therein a processing ofcombining a user body image with a clothes image. Specifically, an imageprocessing server described in Patent Literature 1 changes a size of theclothes image and adjusts an orientation thereof based on body profiledata (such as body height and shoulder width) attached on the user bodyimage and the orientation of the body in the image, and combines it onthe body image.

CITATION LIST Patent Literature

Patent Literature 1: JP 2006-304331A

SUMMARY OF INVENTION Technical Problem

With the try-on image generation technique as described in PatentLiterature 1, however, a clothes image is combined on a user body image,and thus the virtual clothes are always drawn in front of the actualobject (body). For example, also when a user's hand is in front of thetorso, virtual clothes are drawn in front and thus the hand is hiddenbehind the virtual clothes.

As described above, when an anteroposterior relationship between anactual object and a virtual object is complicated, it was difficult toproperly generate a combined image.

Thus, the present disclosure proposes a novel and improved informationprocessing apparatus capable of judging an anteroposterior relationshipbetween an actual object and a virtual object for each part, and adisplay control method and a program.

Solution to Problem

According to an embodiment of the present disclosure, there is providedan information processing apparatus including a judgment unit forjudging an anteroposterior relationship between a shot actual object anda virtual object for each part by use of depth information, and adisplay control unit for displaying a virtual image in which the virtualobject is projected to be overlapped on a shot image in which the actualobject is shot based on the anteroposterior relationship judged by thejudgment unit.

According to an embodiment of the present disclosure, there is provideda display control method comprising the steps of judging ananteroposterior relationship between a shot actual object and a virtualobject for each part by use of depth information, and displaying avirtual image in which the virtual object is projected to be overlappedon a shot image in which the actual object is shot based on theanteroposterior relationship judged in the judgment step.

According to an embodiment of the present disclosure, there is provideda program for causing a computer to perform a processing of judging ananteroposterior relationship between a shot actual object and a virtualobject for each part by use of depth information, and a processing ofdisplaying a virtual image in which the virtual object is projected tobe overlapped on a shot image in which the actual object is shot basedon the anteroposterior relationship judged by the judgment processing.

Advantageous Effects of Invention

As described above, according to the present disclosure, it is possibleto judge an anteroposterior relationship between an actual object and avirtual object for each part.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining the outline of an AR try-on systemaccording to one embodiment of the present disclosure.

FIG. 2 is a diagram for explaining an exemplary overwritten clothesimage.

FIG. 3 is a diagram for explaining other exemplary overwritten clothesimage.

FIG. 4 is a block diagram illustrating a structure of an informationprocessing apparatus according to one embodiment of the presentdisclosure.

FIG. 5 is a diagram for explaining a positional relationship between acamera and a subject in a real space, and a shot image shooting thesubject therein.

FIG. 6 is a diagram for explaining skeleton information according to oneembodiment of the present disclosure.

FIG. 7 is a diagram for explaining depth information according to oneembodiment of the present disclosure.

FIG. 8 is a diagram for explaining a positional relationship between avirtual camera and virtual clothes in a virtual space, and a virtualclothes image projecting the virtual clothes therein.

FIG. 9 is a flowchart illustrating basic display control processings inAR try-on image display according to one embodiment of the presentdisclosure.

FIG. 10 is a flowchart illustrating AR try-on image drawing controlbased on depth information according to one embodiment of the presentdisclosure.

FIG. 11 is a diagram for explaining an exemplary drawn AR try-on imageaccording to one embodiment of the present disclosure.

FIG. 12 is a diagram for explaining an exemplary drawn AR try-on imageaccording to one embodiment of the present disclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a preferred embodiment of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the drawings, elements that have substantiallythe same function and structure are denoted with the same referencesigns, and repeated explanation is omitted.

The explanation will be made in the following order.

1. Outline of AR try-on system according to one embodiment of thepresent disclosure

2. Structure of information processing apparatus

3. Display control

3-1. Basic display control

3-2. Drawing of AR try-on image in anteroposterior relationship

4. Conclusion

1. OUTLINE OF AR TRY-ON SYSTEM ACCORDING TO ONE EMBODIMENT OF THEPRESENT DISCLOSURE

In recent years, a technique called augmented reality (AR) foroverlapping additional information on the real world and presenting itto a user is paid attention. Information to be presented to the user inthe AR technique can be visualized by use of various forms of virtualobjects such as text, icon or animation. One main field to which the ARtechnique is applied is to support user's activities in the real world.In the following, the AR technique is applied to a try-on system.

With the try-on system utilizing the AR technique, a virtual clothesimage is displayed to be overlapped in cooperation of user's activitiesso that virtual try-on can be experienced in real-time. The AR try-onsystem according to one embodiment of the present disclosure judges ananteroposterior relationship between a subject which is a real objectand virtual clothes which are a virtual object, and displays the virtualimage to be overlapped based on the judged anteroposterior relationship.

The outline of the AR try-on system according to one embodiment of thepresent disclosure will be described with reference to FIG. 1. Asillustrated in FIG. 1, the AR try-on system 1 according to oneembodiment of the present disclosure has an information processingapparatus 10, a camera 15, a sensor 17 and a display device 19. A placewhere the AR try-on system 1 is installed is not particularly limited.For example, the AR try-on system 1 may be installed at user's home ormay be installed at a shop.

In the example illustrated in FIG. 1, a plurality of units (theinformation processing apparatus 10, the camera 15, the sensor 17 andthe display device 19) configuring the AR try-on system 1 are separatelyconfigured, but the structure of the AR try-on system 1 according to thepresent disclosure is not limited thereto. A combination of some of theunits configuring the AR try-on system 1 may be integrated. For example,a plurality of units configuring the AR try-on system 1 may beincorporated in Smartphone, PDA (Personal Digital Assistants), cellphone, portable music player, portable video processor or portable gameplayer.

The camera 15 (shooting device) shoots objects present in a real space.The objects present in a real space are not particularly limited, andmay be creatures such as humans and animals, or any but creatures suchas garage or TV stand. In the example illustrated in FIG. 1, a subject A(person, for example) as a real-space object is shot by the camera 15.An image shot by the camera 15 (which may be called shot image) isdisplayed on the display device 19. The shot image displayed on thedisplay device 19 may be a RGB image. The camera 15 sends the shot imageto the information processing apparatus 10.

The sensor 17 has a function of detecting parameters from the realspace, and sends the detection data to the information processingapparatus 10. For example, when the sensor 17 is configured of aninfrared sensor, the sensor 17 can detect infrared rays from the realspace and supply electric signals based on the infrared rays asdetection data to the information processing apparatus 10. Theinformation processing apparatus 10 can recognize real-space objectsbased on the detection data, for example. The type of the sensor 17 isnot limited to an infrared sensor. In the example illustrated in FIG. 1,it is assumed that the detection data is supplied from the sensor 17 tothe information processing apparatus 10, but the detection data to besupplied to the information processing apparatus 10 may be an image shotby the camera 15.

The information processing apparatus 10 combines virtual objects with ashot image or modifies the shot image depending on a recognition resultfor real-space objects, thereby processing the shot image. The displaydevice 19 can display an image processed by the information processingapparatus 10.

For example, as illustrated in FIG. 1, the information processingapparatus 10 can recognize the subject A in the real space, and displaya try-on image in which a clothes image is combined on the displaydevice 19 in real-time. Herein, the user body is a real-space video, andan image of clothes to be tried on is a virtual object displayed to beoverlapped on the real-space video. Thereby, the AR try-on system 1 canprovide virtual try-on in real-time.

Herein, with the virtual try-on system, a clothes image is overwrittenon a shot image shooting a subject therein, and thus the virtual object(virtual clothes) is always drawn in front of the actual object(subject). Bad effects caused by the overwriting of a clothes image willbe described below with reference to FIG. 2 and FIG. 3.

FIG. 2 is a diagram for explaining an exemplary overwritten clothesimage. As illustrated in FIG. 2, if virtual clothes are overwritten onthe subject when a hand of the subject is positioned in front of thetorso, the hand positioned in front of the torso of the subject ishidden behind the virtual clothes and thus an anteroposteriorrelationship therebetween is not correctly expressed.

There is also proposed a method for trying to properly express ananteroposterior relationship by use of color information. For example, aclothes image is not drawn on flesh-colored parts in a shot image.Thereby, when a hand is positioned in front of the torso of the subject,a clothes image is not drawn on the hand. However, when the subject putsnearly flesh-colored clothes on, a clothes image is difficult toproperly draw with the method. As illustrated in FIG. 3, when thesubject puts long-sleeved clothes on, only the hand looks isolated atthe center of the torso with the method.

There is additionally proposed a method for trying to properly expressan anteroposterior relationship by segmenting objects in a shot image.The segmentation is to divide an image into regions in units of objectssuch as “person” and “clothes” and to compare the anteroposteriorrelationships for each object, and thus it is possible to determine that“‘clothes’ are in front of ‘person’” or “‘person’ is in front of‘clothes’.” However, for example, as illustrated in FIG. 2, it isdifficult to consecutively determine that “‘clothes’ are in front of‘person’ and ‘person’ is in front again” when a hand is positioned infront of the torso. Therefore, it is difficult to correctly determine ananteroposterior relationship between an actual object and a virtualobject even with the segmentation.

It was difficult to correctly determine an anteroposterior relationshipbetween each part of one object of “person” such as a hand or the torsodescribed above, and a virtual object. If an anteroposteriorrelationship cannot be correctly determined, consequently virtualclothes cannot be properly drawn at any parts, which has caused areduction in accuracy of the virtual try-on system.

The AR try-on system according to one embodiment of the presentdisclosure determines an anteroposterior relationship between a subjectand virtual clothes for each part by use of depth information, anddisplays a clothes image in an overlapping manner based on thedetermined anteroposterior relationship. Thus, with the AR try-on systemaccording to the present embodiment, for example, as illustrated in FIG.1, even when a hand is positioned in front of the torso of the subjectA, a try-on image having a correct anteroposterior relationship betweenthe subject and the virtual clothes can be displayed on the displaydevice 19.

2. STRUCTURE OF INFORMATION PROCESSING APPARATUS

A structure of the information processing apparatus 10 achieving the ARtry-on system according to the present disclosure will be describedbelow with reference to FIG. 4. As illustrated in FIG. 4, theinformation processing apparatus 10 has a control unit 100, an operationinput unit 120 and a storage unit 130. The control unit 100 has askeleton position calculation unit 101, a display control unit 105, adepth calculation unit 113 and an anteroposterior relationship judgmentunit 115. The information processing apparatus 10 is connected with thecamera 15, the sensor 17 and the display device 19 in a wired orwireless manner.

The control unit 100 corresponds to a processor such as CPU (CentralProcessing Unit) or DSP (Digital Signal Processor). The control unit 100executes a program stored in the storage unit 130 or other storagemedium thereby to operate various functions of the control unit 100described later. All of the respective blocks configuring the controlunit 100 may not be incorporated in the same device and part of them maybe incorporated in other device (such as server).

The storage unit 130 stores programs and data for the processings by theinformation processing apparatus 10 by use of a storage medium such assemiconductor memory or hard disk. For example, it stores a program forcausing a computer to function as the control unit 100. Further, thestorage unit 130 stores data to be used by the control unit 100 therein,for example. The storage unit 130 according to the present embodimentstores 3D data on clothing ornaments as virtual objects to be displayed.In the present specification, the clothing ornaments include clothes andaccessories. The accessories include glasses, hats, belts and the like.

The operation input unit 120 is configured of an input means such asmouse, keyboard, touch panel, button, microphone, switch, lever orremote controller by which the user inputs information, and an inputcontrol circuit for generating an input signal based on user's input andoutputting it into the control unit 100. The user can instruct to powerON/OFF the power supply of the information processing apparatus 10 or toactivate an AR try-on system program by operating the operation inputunit 120.

The camera 15 (shooting device) shoots a real space by use of a shootingdevice such as CCD (Charge Coupled Device) or CMOS (Complementary MetalOxide Semiconductor), thereby generating a shot image. According to theembodiment of the present disclosure, the camera 15 is assumed to beconfigured separately from the information processing apparatus 10, butthe camera 15 may be part of the information processing apparatus 10.

The camera 15 supplies setting information of the camera 15 at the timeof shooting to the information processing apparatus 10. Herein, FIG. 5illustrates a diagram for explaining a positional relationship betweenthe camera 15 and the subject A in a real space, and a shot image A′shooting the subject A therein. In FIG. 5, a focal distance f_(real)between the main point as an optical center of the lens (notillustrated) of the camera 15 and an imaging device (not illustrated) ofthe camera 15, and a shot image A′ (2D, xy coordinate) of the subject A(3D, xyz coordinate) shot on the imaging device are illustrated on theside of the subject for convenience. A distance d_(real) between thecamera 15 and the subject A is calculated as depth information asdescribed later. An angle of view θ_(real) of the camera 15 isdetermined mainly depending on the focal distance f_(real). The camera15 supplies the setting information of the camera 15 such as focaldistance f_(real) (or angle of view θ_(real)) and the number of pixelsof the shot image A′ to the information processing apparatus 10.

The sensor 17 has a function of detecting parameters from a real space.For example, when the sensor 17 is configured of an infrared sensor, thesensor 17 can detect an infrared ray from a real space and supply anelectric signal based on the infrared ray as detection data to theinformation processing apparatus 10. The type of the sensor 17 is notlimited to an infrared sensor. When an image shot by the camera 15 issupplied as detection data to the information processing apparatus 10,the sensor 17 may not be present.

The display device 19 is a display module configured of LCD (LiquidCrystal Display), OLED (Organic light-Emitting Diode) or CRT (CathodeRay Tube). According to the embodiment of the present disclosure, thedisplay device 19 is assumed to be configured separately from theinformation processing apparatus 10, but the display device 19 may bepart of the information processing apparatus 10.

Subsequently, a functional structure of the control unit 100 will bedescribed. As described above, the control unit 100 has the skeletonposition calculation unit 101, the display control unit 105, the depthcalculation unit 113 and the anteroposterior relationship judgment unit115.

(Skeleton Position Calculation Unit 101)

The skeleton position calculation unit 101 calculates skeleton positionsof an object shot in a shot image based on detection data. A method forcalculating real-space skeleton positions of an object shot in a shotimage is not particularly limited. For example, the skeleton positioncalculation unit 101 recognizes a region in which an object is presentin a shot image (which will be also denoted as “object presence region”below) and acquires depth information of the object in the shot imagefrom the depth calculation unit 113. Then, the skeleton positioncalculation unit 101 may recognize the real-space sites (such as head,left shoulder, right shoulder and abdomen) of the object shot in theshot image, and calculate the center position of each site as a skeletonposition based on the depths and shape (characteristic amount) of theobject presence region. The skeleton position calculation unit 101 usesa characteristic amount dictionary stored in the storage unit 130 tocorrelate the characteristic amounts determined from the shot image withthe characteristic amount of each site of the object previouslyregistered in the characteristic amount dictionary, thereby recognizingthe sites of the object contained in the shot image.

Various methods are assumed as a method for recognizing an objectpresence region. For example, when a shot image is supplied as detectiondata to the information processing apparatus 10, the skeleton positioncalculation unit 101 can recognize an object presence region based on adifference value between the shot image in which the object is not shotand the shot image in which the object is shot. More specifically, theskeleton position calculation unit 101 can recognize, as an objectpresence region, a region in which a difference value between the shotimage in which the object is not shot and the shot image in which theobject is shot exceeds a threshold.

For example, when parameters detected by the sensor 17 are supplied asdetection data to the information processing apparatus 10, the skeletonposition calculation unit 101 can recognize an object presence regionbased on the detection data. More specifically, the skeleton positioncalculation unit 101 can recognize, as an object presence region, aregion in which the detected amount of infrared rays exceeds athreshold.

The skeleton position calculation unit 101 recognizes the real-spacesites (such as head and shoulders) of the object shot in the shot imagebased on the depths and shape (characteristic amount) of the objectpresence region acquired with each method described above, andcalculates the coordinate of the skeleton position of each site.Skeleton information containing skeleton positions of one or more sitesconfiguring the subject A calculated in the skeleton positioncalculation unit 101 will be described below with reference to FIG. 6.

FIG. 6 is a diagram for explaining skeleton information. In the exampleillustrated in FIG. 6, the skeleton information is indicated with thecoordinates B1 to B3, B6, B7, B9, B12, B13, B15, B17, B18, B20 to B22,and B24 indicating the positions of 15 sites configuring the subject A,but the number of sites contained in the skeleton information is notparticularly limited.

The coordinate B1 indicates a coordinate of “Head”, the coordinate B2indicates a coordinate of “Neck”, the coordinate B3 indicates acoordinate of “Torso”, the coordinate B6 indicates a coordinate of“Right Shoulder” and the coordinate B7 indicates a coordinate of “RightElbow.” The coordinate B9 indicates a coordinate of “Right Hand”, thecoordinate B12 indicates a coordinate of “Left Shoulder”, the coordinateB13 indicates a coordinate of “Left Elbow”, and the coordinate B15indicates a coordinate of “Left Hand.”

The Coordinate B17 indicates a coordinate of “Right Hip”, the coordinateB18 indicates a coordinate of “Right Knee”, the coordinate B20 indicatesa coordinate of “Right Foot”, and the coordinate B21 indicates acoordinate of “left Hip.” The coordinate B22 indicates a coordinate of“Left Knee” and the coordinate B24 indicates a coordinate of “LeftFoot.”

(Depth Calculation Unit 113)

The depth calculation unit 113 calculates depth information of an objectin a shot image based on detection data. A method for calculating areal-space depth of the object shot in the shot image (herein, adistance from the camera 15) is not particularly limited, and variousmethods are assumed therefor.

For example, the depth calculation unit 113 can calculate depthinformation of an object in a shot image based on the parametersdetected by the sensor 17. More specifically, when a light such asinfrared ray is irradiated on an object from an irradiation device (notillustrated), the depth calculation unit 113 analyzes the light detectedby the sensor 17 thereby to calculate depth information of the object inthe shot image.

For example, the depth calculation unit 113 can calculate depthinformation of an object in a shot image based on a phase delay of thelight detected by the sensor 17. The method may be called TOF (Time OfFlight) system. Alternatively, when a light irradiated from anirradiation device (not illustrated) is configured of a well-knownpattern, the depth calculation unit 113 analyzes a distortion of thepattern configuring the light detected by the sensor 17, thereby tocalculate depth information of the object in the shot image.

The depth calculation unit 113 provided in the control unit 100 has beendescribed above. Herein, the imaging device having a function ofcalculating depth information of an object in a shot image is calleddepth camera, and can be realized by a stereo camera or laser rangescanner. When the information processing apparatus 10 can acquire depthinformation from an external device such as depth camera, the depthcalculation unit 113 may not be present.

Depth information calculated by the depth calculation unit 113 oracquired from a depth camera or the like will be described below. Thedepth information is acquired in an actual dimension (in centimeters)per pixel of a shot image, for example. FIG. 7 is a diagram in whichsuch depth information is expressed in an image. In the imageillustrated in FIG. 7, a degree of depth is indicated in a gray scaledmanner. Specifically, a higher degree of depth (a longer distance fromthe camera 15) is indicated as being whiter and a lower degree of depth(closer distance from the camera 15) is indicated as being blacker. Inthis way, the depth information in the shot image can be acquired in anactual dimension (in centimeters) per pixel.

(Anteroposterior Relationship Judgment Unit 115)

The anteroposterior relationship judgment unit 115 judges ananteroposterior relationship between a shot actual object and a virtualobject for each part by use of depth information.

More specifically, for example, the anteroposterior relationshipjudgment unit 115 compares depth information of an object in a shotimage (see the distance d_(real) indicated in FIG. 5) with depthinformation of an object in a virtual image (see the distanced_(vertual) illustrated in FIG. 8), thereby judging an anteroposteriorrelationship per pixel.

The anteroposterior relationship judgment unit 115 may acquire the depthinformation of the object in the shot image from the depth calculationunit 113 and acquire the depth information of the object in the virtualimage from the display control unit 105.

Herein, it can be assumed that as the depth is smaller (shallower), thedistance between the camera and the object is shorter. It can be assumedthat as the depth is larger (deeper), the distance between the cameraand the object is longer. Thus, the anteroposterior relationshipjudgment unit 115 according to the present embodiment compares the depthinformation of the object in the shot image with the depth informationof the object in the virtual image, and judges that the object having asmaller depth is positioned in front and the object having a largerdepth is positioned behind. Further, the anteroposterior relationshipjudgment unit 115 outputs the judgment result to the display controlunit 105.

(Display Control Unit 105)

The display control unit 105 generates an AR try-on image in whichvirtual clothes are displayed to be overlapped on a subject shot in ashot image, and displays it on the display device 19. The displaycontrol unit 105 according to the present embodiment displays a virtualimage to be overlapped based on an anteroposterior relationship for eachpart between the object (such as the subject A) in the shot image andthe virtual object (such as virtual clothes C) judged by theanteroposterior relationship judgment unit 115. Thereby, the displaycontrol unit 105 according to the present embodiment can control todisplay a correct AR try-on image even when the anteroposteriorrelationship between the subject A and the virtual clothes C iscomplicated.

Generation of a virtual image to be overlapped on a shot image will bedescribed below with reference to FIG. 8. FIG. 8 is a diagram forexplaining a positional relationship between a virtual camera 25 and thevirtual clothes C in a virtual space, and a virtual clothes image C′(also called virtual image) in which the virtual clothes C are projected(rendered). In FIG. 8, the rendered virtual clothes image C′ isillustrated on the side of the virtual clothes similarly to the shotimage A′ shooting a real space therein illustrated in FIG. 5

The setting (internal parameters) of the virtual camera 25 is determinedaccording to the setting (internal parameters) of the camera 15 forshooting a real space. The setting (internal parameters) of the camerais a focal distance f, an angle of view θ, the number of pixels, and thelike, for example. The display control unit 105 sets the virtual camera25 to match with the real-space camera 15 (which may be calledinitialization).

Then, the display control unit 105 arranges the virtual clothes Cadjusted for the skeleton positions of the subject at a position distantfrom the virtual camera 25 by the same distance d_(vertual) as thereal-space distance d_(real) between the camera 15 and the subject Abased on the depth information of the object in the shot image. Thedisplay control unit 105 may generate the virtual clothes C based onpreviously-modeled 3D data. For example, as illustrated in FIG. 8, thesurface of the virtual clothes C are configured in a set of triangles sothat the display control unit 105 can express the 3D shape of thevirtual clothes more realistically. When the skeleton positions of thesubject A change over time, the display control unit 105 can change thearrangement of the virtual clothes C for tracking the skeletonpositions.

The rendered or 3D clothes image C is projected onto a 2D plan image bythe virtual camera 25 so that the display control unit 105 acquires thevirtual clothes image C′ (virtual image).

When displaying the clothes image C′ to be overlapped on the shot imageA′ (see FIG. 5), the display control unit 105 controls the presence ofthe drawn clothes image C′ based on the judgment result by theanteroposterior relationship judgment unit 115 for each part (such aspixel). The display control by the display control unit 105 will bedescribed in the following <3. Display control> in detail.

The structure of the information processing apparatus 10 achieving theAR try-on system according to one embodiment of the present disclosurehas been described above in detail. Subsequently, the display controlfor an AR try-on image by the information processing apparatus 10 willbe described.

3. DISPLAY CONTROL 3-1. Basic Display Control

FIG. 9 is a flowchart illustrating basic display control processings onan AR try-on image by the information processing apparatus 10. Asillustrated in FIG. 9, at first, in step S110, the display control unit105 initializes the setting of the virtual camera 25 in a virtual spaceto match with the setting of the camera 15 in a real space.

Then, in step S113, the skeleton position calculation unit 101calculates skeleton positions (xyz coordinates) of the subject A in ashot real space, and outputs it to the display control unit 105.

Next, in step S116, the display control unit 105 arranges the virtualclothes C to be adjusted to the skeleton positions (xyz coordinates) ofthe subject A in the virtual space.

Then, in step S119, the display control unit 105 renders the virtualclothes C to acquire a clothes image C′ (virtual image), draws an ARtry-on image in which the clothes image C′ is overlapped on the shotimage A′, and displays it on the display device 19 (AR display control).

The information processing apparatus 10 repeatedly performs steps S113to S119 in step S122 until an end instruction is made. Thereby, theinformation processing apparatus 10 can provide an AR try-on imagetracking the motions of the subject A in real-time.

The basic display control processings have been described above. Theinformation processing apparatus 10 according to the present embodimentcontrols to draw an AR try-on image based on the anteroposteriorrelationship between the subject A and the virtual clothes C in stepS119. The drawing of an AR try-on image in an anteroposteriorrelationship according to the present embodiment will be specificallydescribed below with reference to FIG. 10.

3-2. Drawing of AR Try-on Image in Anteroposterior Relationship

FIG. 10 is a flowchart illustrating AR try-on image drawing controlprocessings based on depth information by the information processingapparatus 10 according to the present embodiment. More specifically,FIG. 10 illustrates that the display control unit 105 controls thepresence of the drawn clothes image C′ displayed to be overlapped on theshot image A′ based on a judgment result by the anteroposteriorrelationship judgment unit 115 for each part (such as pixel) in thedisplay control in step S119 illustrated in FIG. 9.

At first, in step S125 in FIG. 10, the display control unit 105calculates a coordinate p (x, y) of the virtual image C′ based on a 3Dcoordinate P (x, y, z) of the virtual clothes C as illustrated in FIG.8. For example, the display control unit 105 multiplies the 3Dcoordinate P (x, y, z) by a projection matrix found from the innerparameters of the camera 15, thereby calculating the 2D coordinate p (x,y) on the image plane.

Then, in step S127, the display control unit 105 calculates a distanced_(vertial) in the virtual space between the virtual camera 25 and thecoordinate P (x, y, z) of the virtual clothes C as illustrated in FIG.8. The display control unit 105 outputs the calculated 2D coordinate p(x, y) and distance d_(vertual) to the anteroposterior relationshipjudgment unit 115.

Then, in step S128, the anteroposterior relationship judgment unit 115acquires a real-space depth (see the distance d_(real) illustrated inFIG. 5) at a 2D coordinate o (x, y) of the shot image A′ correspondingto the 2D coordinate p (x, y) of the clothes image C′ (virtual image).Specifically, for example, the depth at the 2D coordinate o (x, y) ofthe shot image A′ corresponding to the 2D coordinate p of the clothesimage C′ is acquired based on the depth information per pixel (2Dcoordinate) as illustrated in FIG. 7 calculate by the depth calculationunit 113. Herein, as illustrated in FIG. 5, a distance d_(real) betweenthe camera 15 and the 3D coordinate O (x, y, z) of the subject A isacquired.

Subsequently, in step S131, the anteroposterior relationship judgmentunit 115 compares the distance d_(vertual) in the virtual space and thedistance d_(real) in the real space. When the distance d_(vertual) isshorter than the distance d_(real), the anteroposterior relationshipjudgment unit 115 judges that the virtual clothes C are positioned infront of (closer to) the subject A, and the processing proceeds to stepS134. When the distance d_(vertual) is longer than the distanced_(real), the anteroposterior relationship judgment unit 115 judges thatthe virtual clothes C are positioned behind (far away from) the subjectA, and the processing proceeds to step S137. In this way, theanteroposterior relationship judgment unit 115 can judge ananteroposterior relationship between the subject A and the virtualclothes C for each part (per coordinate).

Then, in step S134, when the anteroposterior relationship judgment unit115 judges that the virtual clothes C are positioned in front of thesubject A, the display control unit 105 draws the pixel at thecoordinate p of the virtual image C′ on the pixel at the coordinate o ofthe shot image A′.

On the other hand, in step S137, when the anteroposterior relationshipjudgment unit 115 judges that the virtual clothes C are positionedbehind the subject A, the display control unit 105 does not draw thepixel at the coordinate p of the virtual image C′ on the pixel at thecoordinate o of the shot image A′.

The processings in steps S125 to S137 described above are performed forall the pixels (coordinates) of the clothes image C′ displayed to beoverlapped on the shot image A′. Thereby, even when the anteroposteriorrelationship between the subject A and the virtual clothes C iscomplicated, the display control unit 105 can correctly draw an ARtry-on image. An exemplary drawn AR try-on image by the display controlunit 105 according to the present embodiment

(Exemplary Drawn AR Try-on Image)

Subsequently, exemplary drawn AR try-on images by the display controlunit 105 according to the present embodiment will be illustrated in FIG.11 and FIG. 12.

As illustrated in FIG. 11, even when a hand of the subject A ispositioned in front of the torso, an anteroposterior relationship isjudged for each part when the virtual clothes C are displayed to beoverlapped (combined), and thus the virtual clothes C are not drawn onthe hand of the subject A positioned in front of the torso unlike thegeneral combination illustrated in FIG. 2.

As illustrated in FIG. 12, even when the subject A puts long-sleevedclothes on and a hand is positioned in front of the torso, ananteroposterior relationship is judged for each part according to thepresent embodiment and thus the virtual clothes C are not drawn on thehand of the subject A positioned in front of the torso, unlike thecombination based on color information described above in FIG. 3.

4. CONCLUSION

As described above, with the AR try-on system according to oneembodiment of the present disclosure, the depth of the subject A in theshot image is compared with the depth of the virtual clothes C in thevirtual image for each part, thereby judging an anteroposteriorrelationship between the subject A and the virtual clothes C for eachpart. The AR try-on system according to the present embodiment can drawan AR try-on image in which the anteroposterior relationship between thesubject A and the virtual clothes C is correctly expressed based on thejudgment result.

The preferred embodiment of the present disclosure has been describedabove with reference to the accompanying drawings, whilst the presentdisclosure is not limited to the above examples, of course. A personskilled in the art may find various alternations and within the scope ofthe appended claims, and it should be understood that they willnaturally come under the technical scope of the present disclosure.

For example, the AR try-on system has been described mainly assumingthat a hand of the subject A is positioned in front of the torso of thesubject A in a complicated anteroposterior relationship, but is notlimited thereto and may be applied to a case in which the subject A hasa bag or the like in front of the body. Further, it can be applied to acase in which an obstacle (such as other person, pole or box) is presentin front of the subject A.

The above AR try-on system has been described above mainly assumingtry-on of virtual clothes, but the objects to be tried on are notlimited to clothes, and may be accessories such as glasses, hat, andbelt.

The above AR try-on system has been described assuming that the subjectis a person, but the subject is not limited to person, and may be ananimal such as dog or cat. In this case, it is possible to provide an ARtry-on system for displaying a pet clothes image to be overlapped on ashot image shooting an animal therein.

The technique according to the present embodiment for judging ananteroposterior relationship between an actual object and a virtualobject for each part by use of depth information and performing ARdisplay control based on the judged anteroposterior relationship is notlimited to the above AR try-on system, and can be of course applied tovarious AR display control.

Additionally, the present technology may also be configured as below.

(1)

An information processing apparatus including:

a judgment unit for judging an anteroposterior relationship between ashot actual object and a virtual object for each part by use of depthinformation; and

a display control unit for displaying a virtual image in which thevirtual object is projected to be overlapped on a shot image in whichthe actual object is shot based on the anteroposterior relationshipjudged by the judgment unit.

(2)

The information processing apparatus according to (1),

wherein the judgment unit compares a depth of the actual object with adepth of the virtual object, and judges an anteroposterior relationshipdepending on the magnitudes of the depths.

(3)

The information processing apparatus according to (1) or (2), including:

a depth calculation unit for calculating a depth of a shot subject basedon detection data detected from a real space,

wherein the judgment unit compares a depth of the subject calculated bythe depth calculation unit with a depth of a virtual clothing ornamentthereby to judge an anteroposterior therebetween for each part, and

wherein the display control unit displays a virtual clothing ornamentimage in which the clothing ornament is projected to be overlapped on ashot image in which the subject is shot based on the anteroposteriorrelationship judged by the judgment unit.

(4)

The information processing apparatus according to (3),

wherein the display control unit controls the presence of the drawnvirtual clothing ornament image per pixel based on the judgment resultof the anteroposterior relationship per pixel by the judgment unit.

(5)

A display control method including the steps of:

judging an anteroposterior relationship between a shot actual object anda virtual object for each part by use of depth information; and

displaying a virtual image in which the virtual object is projected tobe overlapped on a shot image in which the actual object is shot basedon the anteroposterior relationship judged in the judgment step.

(6)

A program for causing a computer to perform:

a processing of judging an anteroposterior relationship between a shotactual object and a virtual object for each part by use of depthinformation; and

a processing of displaying a virtual image in which the virtual objectis projected to be overlapped on a shot image in which the actual objectis shot based on the anteroposterior relationship judged by the judgmentprocessing.

(7)

The program according to (6),

wherein the judgment processing compares a depth of the actual objectwith a depth of the virtual object thereby to judge an anteroposteriorrelationship depending on the magnitudes of the depths.

(8)

The program according to (6) or (7), for causing the computer to furtherperform a processing of calculating a depth of a shot subject based ondetection data detected from a real space,

wherein the judgment processing compares a depth of the subjectcalculated by the processing of calculating a depth with a depth of avirtual clothing ornament thereby to judge an anteroposteriorrelationship for each part, and

wherein the display processing displays a virtual clothing ornamentimage in which the clothing ornament is projected to be overlapped on ashot image in which the subject is shot based on the anteroposteriorrelationship judged by the judgment processing.

(9)

The program according to (8),

wherein the display processing controls the presence of the drawnvirtual clothing ornament image per pixel based on the judgment resultof the anteroposterior relationship per pixel by the judgmentprocessing.

REFERENCE SIGNS LIST

-   10 information processing apparatus-   15 camera-   17 sensor-   19 display device-   100 control unit-   101 skeleton position calculation unit-   105 display control unit-   113 depth calculation unit-   115 anteroposterior relationship judgment unit-   120 operation input unit-   130 storage unit-   a subject-   b coordinate (skeleton position)-   c virtual clothes

1-9. (canceled)
 10. An information processing apparatus comprising:circuitry configured to: acquire actual object depth information ofdifferent portions of an actual object based on a distance of respectiveportions of the actual object from an object imaging device; determinevirtual object depth information of different portions of a virtualobject based on a distance of the respective portions of the virtualobject from a virtual imaging device at a location corresponding to theobject imaging device; determine an anteroposterior relationship betweenthe actual object within a shot image obtained by the object imagingdevice and the virtual object for each of the portions of the actualobject and corresponding portions of the virtual object by comparingvalues of the actual object depth information and the virtual objectdepth information of the portions; and control a display to display theportions of the virtual object when the portions of the virtual objectare determined to be located in front of the actual object or displaythe portions of the shot image when the portions of the virtual objectare determined to be located behind the actual object so that theobjects are displayed in accordance with the determined anteroposteriorrelationship.
 11. The information processing apparatus of claim 10wherein the actual object is a part of body of a human.
 12. Theinformation processing apparatus of claim 11 wherein the actual objectis a hand.
 13. The information processing apparatus of claim 10 whereinthe anteroposterior relationship is determined on the basis of pixels ofthe actual object and pixels of the virtual object.
 14. The informationprocessing apparatus of claim 10 wherein the display is controlled onthe basis of pixels of the actual object and pixels of the virtualobject.
 15. The information processing apparatus of claim 10 wherein thedisplay is controlled to display the portions of the virtual object whenthe portions of the virtual object are determined to be closer to thevirtual imaging device than the distance of the portions of the actualobject from the object imaging device.
 16. The information processingapparatus of claim 10 wherein the display is controlled to display theportions of the shot image when the portions of the actual objectcorresponding to the portions of the shot image are determined to becloser to the object imaging device than the distance of the portions ofthe virtual object from the virtual imaging device.
 17. A displaycontrol method comprising the steps of: acquiring actual object depthinformation of different portions of an actual object based on adistance of respective portions of the actual object from an objectimaging device; determining virtual object depth information ofdifferent portions of a virtual object based on a distance of therespective portions of the virtual object from a virtual imaging deviceat a location corresponding to the object imaging device; determining ananteroposterior relationship between the actual object within a shotimage obtained by the object imaging device and the virtual object foreach of the portions of the actual object and corresponding portions ofthe virtual object by comparing values of the actual object depthinformation and the virtual object depth information of the portions;and displaying the portions of the virtual object when the portions ofthe virtual object are determined to be located in front of the actualobject or displaying the portions of the shot image when the portions ofthe virtual object are determined to be located behind the actual objectso that the objects are displayed in accordance with the determinedanteroposterior relationship.
 18. A non-transitory computer readablemedium on which is stored a program for causing a computer to control adisplay by: acquiring actual object depth information of differentportions of an actual object based on a distance of respective portionsof the actual object from an object imaging device; determining virtualobject depth information of different portions of a virtual object basedon a distance of the respective portions of the virtual object from avirtual imaging device at a location corresponding to the object imagingdevice; determining an anteroposterior relationship between the actualobject within a shot image obtained by the object imaging device and thevirtual object for each of the portions of the actual object andcorresponding portions of the virtual object by comparing values of theactual object depth information and the virtual object depth informationof the portions; and displaying the portions of the virtual object whenthe portions of the virtual object are determined to be located in frontof the actual object or displaying the portions of the shot image whenthe portions of the virtual object are determined to be located behindthe actual object so that the objects are displayed in accordance withthe determined anteroposterior relationship.